Approximation with neural networks activated by ramp sigmoids

Accurate and parsimonious approximations for indicator functions of d-dimensional balls and related functions are given using level sets associated with the thresholding of a linear combination of ramp sigmoid activation functions. In neural network terminology, we are using a single-hidden-layer perceptron network implementing the ramp sigmoid activation function to approximate the indicator of a ball. In order to have a relative accuracy , we use T = c(d2/ 2) ramp sigmoids, a result comparable to that of Cheang and Barron (2000) [4], where unit step activation functions are used instead. The result is then applied to functions that have variation V f with respect to a class of ellipsoids. Two-hidden-layer feedforward neural nets with ramp sigmoid activation functions are used to approximate such functions. The approximation error is shown to be bounded by a constant times V f /T 1 2 1 + V f d/T 1 4 2 , where T1 is the number of nodes in the outer layer and T2 is the number of nodes in the inner layer of the approximation fT1,T2 . c © 2010 Elsevier Inc. All rights reserved.